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Abstract 

The precision of item parameter estirates can be increased 
by taking advantage of dependencies between the latent proficiency 
variable and auxiliary examinee variables such as age, courses 
taken, and years of schooling. Gains roughly equivalent to two to 
six additional item responses can be expected in typical 
educational and psychological applications. Empirical Bayes 
computational procedures are presented, and illustrated with data 
from the ^r^flla of American Youth survey. 

Key words: EM-algorithm, empirical Bayes, marginal maximum 
likelihood 
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Exploiting Auxiliary Information about Examinees In the 
Estimation of Item Parameters 
A pervasive problem in item response theory (IRT) is the 
difficulty of simultaneously estimating large numbers of parameters 
from limited data. Even large samples of examinees may not 
eliminate the problem when each examinee responds to only a few 
items ^ as in educational assessment and adaptive testing. Certain 
improvements are obtained by using hierarchlal models along the 
lines of Lindley and Smith (1972); treating examinee parameters as a 
sample from a cbmmbh population enhances the stability and precision 
of item parameter as well as examinee parameter estimates. This 
approach has been applied to IRT by a number of researchers 
recently^ including Bock arid Aitkin (1981) ^ Lebriard arid Nbvick 
(1985), Rlgdon and Tsutakawa (1982), and Swaminathan and Giffbrd 
(1982). 

For the most part, the afbrementibriai writers cbrisider ail 
examinees to be members of a single^ undifferentiated^ population. 
This framework instantiates such beliefs as, "if the parameters 
of most examinees seem to lie between -3 and +3, then the 
parameter bf an examinee who answered both of two hard math items 
correctly is probably somewhere between +li5 to +3.5 — even though 
his/her maximum likelihood estimate is -Ho." Additional stability 
and precision may yet be achieved if auxiliary information is 

ERIC 
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available about examinees, such as educational background or status 
on demographic variables. A statement like "the parameter of an 
examinee who answered both of two hard math items correctly and 
studied calculus in college is probably between +2.7 and +3.7," 
might result. 

This paper addresses the utilization of auxiliary information 
abbi't examinees In estimating item parameters. The following 
section reviews Itein parameter estimation when examinee parameters 
are known ^ then when examinee parameters are unKnown and nothing 
is assumed about them. Attention then turns to the additional 
assumptions of first, an undifferentiated population, atid second, 
a population differentiated with respect to auxiliary variables. 
Following this are sections that discuss anticipated gains in 
precision, outline computational procedures, and illustrate the 
approach with responses to four items from the Arithmetic Knowledge 
subtest of the Armed Services Vocational Aptitude Battery. 

The Role of Auxiliary Information 
The relevance of auxiliary examinee variables to item 
parameter estimation Is not immediately obvious, since they play 
no role in the basic model for item responses. Letting x^ « ^^il> 
***'^lh ^^P^^^eht the responses of examinee i to n test itemis and 
y^ represent values of auxiliary variables such as educational and 
demographic states, the standard IRT assumption of local 

7 
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p(xj|ej,yj*B) -n p(xyje£,3j) , (i) 



where 6^ is the examinee parameter ^ g » (B||...i3-) are possibly 
vector-valued item parameters ^ and the form of p(x^^ |9 .B^ ) is 
specified a priori through the item response model. It follows that 
y^^ would indeed be irrelevant to item parameter estimation if 
were known. The likelihood to be maximized with respect to g, 
given the data matrix X = (xj,.i.,Xj^) of responses from N examinees 
with proficiencies 8 = (ejs.».,0j^) and auxiliary variables Y « 
(yj , . . . ,yj^) , would be simply 



N 

fc = II P(x. |e. .B) . (2) 

i --^ - 



The maximum likelihood estimate (MLE) B would satisfy the 
likelihood equations 



0 = E 3£.(0.)/3B . (3) 
1 - 



where £^(6) = log p(xj|0,B)» and the covarlance matrix of 
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estimation error variances for 0 could be approximated by the 



inverse of the observed information matrix Is 



1 




3p' 



(4) 



But Equation 1 gives response probabilities cbriditibned on g. 



wfiere Fj(e) is the distribution of the unknown proficiency of 
examinee i. This is atl "incomplete data" probim, in the 
tenniholbf?y of Drapster, Laird, and Rubin (1977), corresponding 
to the "complete data" problem of maximizing Equation 2 when e 
is known. Assuming the required Integrals exist, the likelihood 
equations become 



and e is not krown in practice. The problem that must actually be 



solved is to maximize the marginal likelihood 



* n / p(xj|e,B) dF.(e) 



(5) 



0 - E (x^) / t3£^(:e)/3p] dFj(e) 



where 
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Louis (1982) shows that if Zack's (1971, Chapter 5) regularity 
conditions are met and if is known for all i, the diagonal 
elements of the incomplete-*data observed information matrix, 
namely 



cannot exceed the diagonal elements • In oth^ r words, the 
precision with with elements of g would be estimateJ if 0 were known 
provides ah upper limit to the precision to be expected when 6 is 
not known but must be inferred. 

A similar phenomenon ariges in the context of sample survey 
analysis when a clustered sajnpling design is employed to estimate 
a mean. If n units are sampled from each of N randomly-selected 
clusters, then the squared standard error of the mean, ignoring 
finite population corrections, is given as 




3ii(e) 3£i(e) 




) ^ } dF^(e) , (6) 
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SEM^ - if U + (n - Dp] , 

where o is the population variance and p is the intraciass 
correlat on coetficieht indicating withih-cluster homogeneityi If 
the number of clusters (N) is held constant^ increasing the sample 
size (h) within clusters cannot decrease SEM below pa /N, the 

_ : z 1-^ 2 ------ _ 

the value of SEM obtained when the means of the sampled clusters 
are kiibwn wJ thbut error • 

The estimatibn of p in the context of IRT must also deal with 
uncertainty from two sources. First is the usual limitation of 
having data from only a finite sample of examinees. All other 
conditions remaining unchanged, increasing N leads to greater 
precision for Second is the limitation that 6 remains unknown 
even for sampled examinees. For a fixed sample of examinees, 
reducing uncertainty about 6 leads to greater precision for 
This can be achieved through (i) item respdhses, (ii) assumptions 
about the F^'s and (iii) auxiliary variables related to 6* 

de Leeuw and Verhelst (1984) point out that finding maxima in 
terms of p and of each individual 9^ in the manner suggested by 
Bimbaum (1968) is equivalent to maximizing Equation 5 when each 
Fj concentrates its mass at the single (unknown) point 6^. This 
joint maximum likelihood (JML) solution utilizes only information 
tn responses x^^ from examinee i to reduce uncertainty about Bj. 
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Alternatively i one may consider the g's to be identicaiiy 

distributed i so rhat « F for all i. An auxiliary variable y 

is thereby implied for all exatnirieesj an indicator signifying that 

each is a member of the population whose distribution is specified 

by F. Appearing in the literature are treatments that assume a 

completely specified form for F (e.g., Bock & Liebermari, 1970), 

others that assume parametric forms with unknown parameters ct to 

be estimated along with g (e.g., Zwarts & Veldhuesen, 1985), and 

still others that provide nonparametric approximations (e.g., 

Tjur, 1982), Under the first of these three approaches, the 

assumed population distribution combines with x^ to produce 

p(0^|X), which in this case equals p(ejjxj). Under the latter 

two approaches i responses from examinees other than examinee i 

also play a role in estimating F so that p(0^|x^) ^ p(0^|x). 

A third alternative, falling between unique^ unconstrained 

F^'s and identical F^'s^ is to posit distributions that depend on 

auxiliary variables: that is, F.(6) « F (0). Examinees with 

^^^^^^ /i 
identical y values are considered a random sample from a 

population indexed by lhat particular value of y, and these 

conditional distributidhs are allowed to vary with yi A following 

section gives details for two special cases, namely a linear model 

and a (quasi--) nonparametric mixture approximation. 
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How Much C^n Be Gained? 

Several factors contribute to the magnitude of the precision 
gains that can be achieved through population assumptions and 
auxiliary variables. One factor is the sensitivity of different 
model parameters tv missing information. Mislevy's (1984) analysis 
of Bock and Lieberman*8 (1970) LSAT data showed that estimates of 
the population variance were more substantially improved by 
increases in test length than were estimates of the population 
meani This might lead one to expect increased information about 6 
to have more effect dri item slopes than on item thresholds in the 
context of item parameter estlniatibh. 

A second factor is the nature of the joint distribution of 
auxiliary variables with 9 . An auxiliary variable adept at 
identifying low proficiency examinees, for example, adds 
information for those ex^inees most useful for estimating lower 
asymptote item parameters^ 

A third factor is the dependence of the estimated Information 
upon estimated parameter values i Although a slope parameter 
may be consistently estimated under both the undifferentiated and 
undifferentiated population models, a higher estimate under the 
latter may appear^ less precise. This is because estimated standard 
errors for slopes are directly proportional to the values of the 
slope estimates, even though true standard errors depend on true 
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slope values atid not their estimates. A slope estimated with the 
aid of atixiliary variables and obtaining a higher estimate can 
thus have a lower true standard error but a higher estimated 
standard error. 

Since the s^e factors determine information gain from both 
increased test length and auxiliary variables, however, it is 
reasonable to consider the contribution of auxiliary variables in 
units of additional item responses. In the special case of 
dichotomous items, the amount of information conveyed by item 
responses alone is 



where Pj(e) = p(xj = ije) and Pj(e) = dP^(e5/de. For examinees 
with finite maximum likelihood estimates, Bayes theorem applied 



with a diffuse prior leads to the approximation p(e|xj^) ^ N(e,0^) 

2 _ -1 

with « i . This follows by first rescalirig the likelihood so 
that it integrates to one, then using its mode and curvature at the 
mode in a normal approximation. 

Consider as an example the two-parameter logistic model, 
under which P. (6) i p(x « ,b.) « i/{l + exp{-1.7a.(e - b.)]}. 




i(e) « E 



p.(e)[i - p,(e)] 



i i 
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The contribution of Item j to information about 6 Is 

z? --- - - __z^ _ Jizz:z_ii 

2i89ajPj(e){i - Pj(e)], and the total information from n identical 

2 

Items for which = 6 and a^ = a is simply 0.7225 na . Table 1 

: : L : a : : : : 

gives values of 1 and 0^ in this simple case for selected test 
lengths and values of a* Note that where l»7a » 1«0 (iie»» 
a * .588 » corresponding to an item trait correlation of .7071 in a 
standard hbrmal population) » four addltldhal items provide a unit 
gain in precision. The results provide an ihdicatidn of the amount 
of information about 6 that is employed in JML estimation of item 
parciineters. It is apparent that as test length ii«crea8es» 
information (i«e.» precision) increases at a constant rate and the 
posterior variance decreases at a decreasing rate* 

Insert Table 1 about here 

The magnitude of aaiti in infortnation about 6 obtained by 
assuming an undifferentiated population (i»e«» B F) can be 
gauged by extending the approximation employed for Table Ik If the 
normalized likelihood function induced by x^ is again approximated 

as N(6 »d^) and if it is further assumed that examinee i has been 

1 z _ - - ... .... 2 z---- 

selected at random from a population in which 6 ^ N(y»0 )» then 

p(6|x^> - N(ei) 



J5 



where 
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and 



" -2 -2 

X ^ 
-2 ^ -2 
^x 



r - (a + ^ 



Table 2 shows values of the reciprocal of I (l.e.^ "precision") 
from various test lengths with identical items with im7a « 1 and a 
standard normal prior for 6. Note that for each test length, a 
unit gain in precision is achieved over the 1.7a « 1 cblutnn of 
Table 1. These tabled values fall within the ranges encountered in 
applied work, and suggest that the assumed distribution contributes 
about as much information about 6 as four additional items. The 
corresponding value for i.7a « i5 is sixteen items ^ and that for 
1.7a « 1.5 is about one item. Since the absolute contribution is 
constant with respect to increasing test length, the r^el^M^ive 
contribution declines. 

To gauge the additional impact of differentiating the 
population through auxiliary variables * we may consider numerical 
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values resulting from a regression model with homoscedastlc 
residuals i Suppose y values account for (100 x z)-percent of the 
variance In a population with total variance 1.0^ so that 
F^(e) - N(p^,a^) with a| « 1 - r. If the normalized likelihood 
induced by item responses is approximately N(6,a^), then 

0^-2 ^ -2 

a + a 
X e 

Using the same simplified item response model and *a* value as 
Table 2^ Table 3 compares values of the inverse of the posterior 
variance for 0 as determined by (i) item responses alone ^ (ii) with 
knowiedge of membership in an undifferentiated population with unit 
variance I and (iii) with the additional knowledge of auxiliary 
variables that account for successively greater proportions of 
total variance. Values between 10- and AO-percent, a range typical 
of educational and psychological wbrk^ increase ihfbrmatidh 
(posterior precision) abottt 0 by amounts roughly equivalent to one 
to three additional item responses. For items with 1.7a « .5, 
gains iti item units would be doubled; for items with l.Va » 1.5, 
gains in item units would be halved. 

Insert Tables 2-3 about here 
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the Ignorability of p(y) 
This section deinonstrates that under reasonable assumptions, 
the population distribution of y can be Ignored for the purposes of 
esttmattng Item parameters B and population parameter 
Suppose that the distribution of y In a population of examinees Is 
governed by the density function p(y|Y)ii which depends oh possibly 
unknown parameters y ^ut not upon item parameters 3 nor on the 
parameters at of the conditional distributions f(6|y,a)« The 
probability of observing the data matrix (X,Y) from a random 
sample of N examinees is given by 

P(X,YjB,a,Y) 



n / p(x^|e .yj^.e.q.Y) p(e |y^,e,q,Y) pCyj^le.a.Y) do 



II / P(x. |e,e) p(e|y ,a) p(y. |y) de 

i - 



- { II / P(x. |e,6) p(e|y. ,a) de} X { n p(y. |y)} 
i -1- ^- i - 



p(x|y3,o) p(y|y) . (6) 



is 
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Likelihood Ihferehceis about a and 3 are therefore Independent of 
inferences about y, and the conditional MLE's of a and g given Y 
are Identical to MLE's obtained jointly with y» 

Models and Methods 

this section presents two IRT models that differentiate 
examinees by means of auxiliary variables » and suggests computing 
apprbximatiohs based oh Bock and Aitkin's (1581) marginal maximum 
likelihood (empirical Bayes) procedures. 
Mixtures of Finite Distributions 

Mlslevy (1984) decribes a nonpar^etrlc approximation of a 

continuous density function of a latent variable in terms of a 

distribution with mass at a finite number of prespecified points. 

the proficiency of each examinee, or 6^, then, is assumed to take 

one of only Q known values, the "latent trait" problem is thereby 

replaced by ah analogous "latent class" problem that is easier to 

solve. A single population was addressed in that presentation, 

and item parameters were assumed known. We now consider extensions 

to the simul'caneous estimation of it^ parameters^ and to multiple 

subpopuiations indexed by an auxiliary variable y. this approach 

provides considerably flexibility in the distributions Fj^(8) « 

F (6). It lends itself well to discrete auxiliary variables with 
7l 

relatively few values. 
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It proves convenient to write siich ah auxiliary variable as 
a vector of 0/1 indicators • Define y^ » ^^ii » • ' • »^ik^ letting 
y.^ » 1 if examinees i is associated with the k'th of K exhaustive 
and mutually exclusive subpbpulatibiis ^ and zero otherwise* The 
probability of observing response pattern x^ from ah examinee 
selected at random from a specified subpopulation is given by 



P(x^|y^,B) « R { / P(x^|e,6) dF^(e)} , (7) 



where Fj^ is the distributlbh ih subpbpuatibh fc* This probability 
can be approximated by a finite distribution as 



-- - """~'xiC 

k q 

where 0ji*»»i0Q is a grid bf pbihts and W-j^ is th^ weight or 
density at point q in subpopulatioh ko The weights W play the 
role of a in earlier notation. For the remainder of this 
subsectibh^ we limit bur attehtibh to distributibhs bf the form 
of the right-hand side of Equation 8* As demonstrated above, we 
may carry out the estimation of 3 and W conditional on 
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Let (X»Y) be the data matrix observed from a sample of N 
examinees selected either randomly from the population as a ^ole 
or as random subsamples svratlfled on y« The probability of X 
given Y is proportional to 



U - 11 n { z p(x. |0„,e)M„a 



^ik 



and its logarithm is 



- E E y.. log E p(x. |e„,B) W^. 

i k ^'^ q -1 q . qK 

Relative maxima with respect to ^ and W can be obtained by means 
of the EM algorithm » under the special case of missing indicators 
for a multinomial distributlbn (Dempster et al«, 1977, Section 
4.3)« The expectation step of cycle t 4- 1 computes expected 
values of the following quantities: 



EKLC 
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The expected number of examinees with proficiency 9^ from a 

sample of size Nj^ from subpopulatlon kj conditional on Y, 

*t ""t" 
g ^ and W*: 



where 



ah appllcatlbh of Bayes theorem^ gives the posterior 
probability that the proficiency of examinee 1 is G^, given 

A ^ ^ t 

provisional parameter estimates 3 and W • 
The expected number of correct responses to item j from 
ex^inees in subpbpulatibn k with proficiency 0^, given a 
random sample of size ^^ (again given 3 and W ): 
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The maximization step computes what would be HLE's of $ and 

- - A - - - A - . — - - .- .- . . ..... . _ _ 

W if N and R were observed quantities rather than conditional 
expectations. For W, we have simply 



-t+i ^ -t+i , 

qk "qk '"k 



For 0, we sol-'e conditional expectations of likelihood equations: 



r I p,(e^)[i - P.ce^)] 3g 



where r!^ i - E R^ , and N is similarly defined. Under the 2- 
jq+ 1^ jqk q* 

parameter logistic model, for example, Equation 9 simplifies as 
follows : 

In principle, the linear indeterminacy in the 2-, and 3- 
parameter logistic and normal IRT models presents no impediment to 
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the EM algorithm^ which readily converges to one of the Infinitely 
many solutions on a ridge. Numerical stability and the quality of 
the finite characterization of F are enhanced, however, by 
cdntrblling the scaling of the solution at this point • One 
convenient way of doing so is to standardize the wighted average 
distribution. We have referred to the points 0_. as specified a 
priori; given the linear indetermihacy , we may conceive of only 
their relative spacing as prespecified. After each m cycle, then, 
we may rescale the points as follows: 

0„ = (e^ - 0)/8 

where 

k k q q qk 

and 

k ^ q ^ qk 
Item parameters are adjusted accordingly* Under the 2- and 3- 
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parameter models, lis replaced by (bj - 0)/8 and a^ is replaced 
by say Under l-parameter models, rescaling takes place only with 
respect to 6* 

Iteration from several starting values helps to verify whether 
a given solution Is Indeed a global maximum i The observed 
Information matrix for the item parameter estimates can then be 
approximated via Equation 6. Employing Louis's (1982) 
simplifications for "missing multinomial indicators" problems, we 
obtain 

^ ^ - (10) 

where P^(0q|xj) is evaluated at 6 and W • 
A Linear Model 

The unrestricted mixture solution described above becomes 
unwieidiy as the number of potential values of the auxiliary 
variable Increases. The more structured alternative of a linear 
model for p(e |y) is suitable when y is vector-valued or is 
continuous rather than discrete* Assuming homoscedastlc and normal 
residuals, we would have 
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9 - N(y'a,a^) 



where auxiliary variables are coded so that the K columns of Y = 
(y. • • y*>)S which are basis vectors for the K elements of a, 
are linearly independent. They may include values oh measured 
variables such as previous test scores and dummy regression 
variables that encode selected contrasts among categorical 
auxiliary variables* 

Maximum likelihood solutions for a and in the special case 
of structured means for the cells of a multi-way design have been 
given by Mislevy (19855 under the assumption that item parameters 
are known, and by Zwarts and Veidhuesen (1985) under the assumption 
that p(x|6) is the Rasch model with unknown item parameters to be 
estimated jointly. These solutions are readily extended to the 
case of a general IRT model with unknown item parameters. This 
section describes an approximation over a grid of prespecified 
points so that computation is similar to the hdhparametric solution 
described above. Attention is focused for convenience upon the 1-, 
2-, and 3-parameter logistic and normal IRT models; 

The linear indetermihacies of these models are again 
conveniently resolved by restrictions on the population parameters. 

- : - 2 

First, we may without loss of generality fix a at unity to set the 
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unit-size of the scale. For 1-parameter models » a slope parameter 
common over items is then estimated. Second , we may set the 
origin by centering the elements of each column of Y at zero. All 
effects are thus cast as deviations around a grand mean of zero. 
This restriction^ in conjunction with the independence of the basis 
vectors y completes the resolution of the scale. 

the marginal likelihood for a sample of size N is written as 

L - n / p(x. |e,B) (|)(e - y^a) de 

i - 

where ^ represents the standard normal density function. 
Approximation over a finite grid of points is accomplished by 

t* - n E P(x^|0_.e) W^.(a) . 
1 q " ^ * M * 

where 

Wqi(a) - expI-<0^ - y^^)2/2]/ E exp[-(0- - y|a)^/2] . 

The weights W play the same role as those in the preceeding 
approximation. The difference is that they are no longer estimated 
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without restriction, but modeled as functions of the effect 
parameters a • 

MML estimation can again proceed in EM cycles that solve the 

......... ^t -t 

likelihood equations. Let 3 and a be provisional estimates from 



cycle t. The E*step computes expected counts of examinees and 
correct responses at each point: 



and 



= t p(o jx. i^a^) 



where 



It also computes the conditional expected value of each examinee's 
proficiency: 
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The M-step pseudo-likelihood equations for item parameters can 
be written as in Equation 9» The equations for a simplify to 



t+1 t+1 t+1 -- '- -1111" :_ 

where 9 ^ •••••^^ posterior information matrix for 



3 can a^^'ain be approximated via Equation 10. 

A Numerical Example 
This section illustrates the procedures described above. The 
data are responses to four items from the Arithmetic Reasoning teat 
of the Armed Services Vocational Aptitude Battery (ASVAB), Form 8A, 
as observed in a sample of 776 participants in the Profile of American 
Youth survey (U.S. Department of Defense* 1982). Table 4 gives 
counts of the sixteen possible response patterns occurring in each 
ceil of a 2-by-2 design based on two background variables collected 
along with item responses. Because these variables are based on 
demographic informatibn rather than the educational ly-relevaht 
Information we would prefer, we will refer to the factors as simply 
Factor A and Factor B, nesting levels 1 and 2 within each. 



Insert Table 4 about here 
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Four analyses were carried oat bit these data* In each, the 
2--pardineter logistic ogive was employed as the IRT model for 
conditional probabilities of correct response* The analyses 
differed in terms of the aiiciliary information about examinees they 
employed* The first run used MML estimation of item parameters and 
densities over a grid of ten points, assuming examinees were drawn 
at random from a single undifferentiated population* The second 
and third runs differentiated the population via Factor A and 
Factor B respectively, and the fourth run employed both factors 
jointly^ 

Resulting item parameter estimates and standard errors, along 
with subpbpulation means and standard deviations, are shown in 
Tables 5 through 8. The scale has been set in all sblutlbns to 
standardize the total population* For each item parameter type, 
columns in Table 6 through 8 display the ratio of the squared 
standard error of the item parameter estimate under the 
undifferentiated model to the corresponding value in the 
differentiated model* The result can be interpreted as efficiency 
relative to the undifferentiated models and the excess bf a value 
above unity reflects the proportional increase in estimation 
precision* Geometric averages are also shown for the relative 
efficiency columns i The excess of such a value over unity, times 
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fbur^ gives the Increases of precision In the units of numbers of 



additional items of the same kind. 



It is apparent that including auxiliary information had little 
effect on the vaiues of the item parameter estimates. The 
differences between the estimates from the undifferentiated and the 
fully differentiated solutions occur only in the second decimal 
place. More significant differences exist in the accbmpahyihg 
(estimated) standard errors > however. The precision of threshold 
estimates was improved only modestly; ah incrvsase roughly 
equivalent to one additional item response per exs^lnee was 
observed in the fully differentiated run. The precision of slope 
estimates was improved dramatically; an increase roughly equivalent 
to eight items was observed- It would appear that Factor A 
accounted for more increase in precision for slopes, while Factor B 
accounted for more increase in precision for thresholds. 

Discussion 

This paper has outlined procedures for incorporating 
auxiliary ihfonhatibn abdiit examinees into the IRT framework. 
Enhancing the precision of item parameter estimates wa$^ the primary 
focus. This section evaluates the value of improvements so 
attained, and discusses two additional aspects of the models 



Insert Tables 5-8 about here 
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The Increase In Information about Item parameters In typical 
educational and psychological settings can be expected to lie In 
the range of two to six Items. The numerical ex^apie suggests that 
the increase will vary by item parameter type, probably less for 
well-estimated parameters and greater for poorly-estimated 
parameters. 

The expected increase is modest, to be sure, but in many 
applications it is free in the sense that it is already available 
for use. Because its incremental value decreases for longer tests, 
auxiliary information would be most useful in settings where 
relatively few response are solicited from each examinee. This 
would include two applications of great current interest ^ namely 
educational assessment and adaptive testing. In assessment, data 
that are sparse at the level of individuals--say ^ five items in a 
given scale — yield more efficient estimates of population 
parameters for a given total number of item responses, in 
adaptive testing^ new items are calibrated using joint response 
patterns with previously-calibrated items while the number of old 
items is held to minimally acceptable levels — as few as, say, 
fifteen. 

A side issue In the present paper but a fundamentally 
important result is that when examinees are indeed a random sample 
from a well-defined population, the estimated population 



32 



Exploiting Auxiliary Information 

30 

distributions and effect parameters are cbhslisteht within the limits 
of precision afforded by the nomericai approximations (see Mislevy, 
i984, i985, on population estimation vheii item parameters are 
known). This stands in contrast to the asyicptbtically biased 
results obtained by using the distribution of 6 to approximate the 
distribution of 6 • Iii fact) the discrepancy between the two 
distributions is largest in exactly those cases in which the present 
procedures offer most the benefit for item parameter estimation, 
namely short tests. 

Finally i it is implicit in preceding discussions that auxiliary 
information about exc^inees can lead to iinprbved estimates of 
individual proficiencies. Whether estimates that are improved in 
the sense of minimum mean squared error are unequivocally "better" 
for all applications is not clear, however. We have avoided 
advocating the use of auxiliary information when tests are used as 
contests— i.e. i when important placement or selection decisions are 
-made for individual examinees— because it would seem that in these 
situations the tester ought to gather enough data directly dependent 
upon proficiency (i.e.^ item repbhses) to make satisfactbrily 
precise decisions on that strength alone. In adaptive testing, for 
example, we would recdinmend the use of auxiliary information to 
improve item parameter estimatibn^ but nbt to estimate scores that 
will be used to compare individual examinees. 
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Posterior Preciision for 9 from 


Item Responses Only 






1.7a 


- ^500 






I. 7a -f 


i.sda 


XI 


i 


2 


i 


-2 
°x 


i 


2 


2 


.125 


8.000 


.500 


2.000 


1.125 


.889 




.250 


4.000 


1*000 


iiOOO 


2i250 


• 444 


8 


.500 


2.000 


2.000 


.500 


4.500 


% ZmZmZm 


16 


1.000 


1.000 


4.000 


.250 


9.000 


.111 


32 


2.000 


s50d 


8i000 


il25 


18i000 


id56 


64 


4.000 


.250 


16.000 


.063 


36.000 


.028 


128 


8.000 


.125 


32.000 


.031 


72.000 


.014 


h 


" number of 


identical 


items with a 


as noted 


and b « 6 . 





i » information = posterior precision. 



.?8 



Exploiting Auxiliary Information 

36 

Table 2 

Posterior Precision for 6 from Item Responses and Population Membership 
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Table 3 



Precision Incr eases for 6 Resulting from the Use of 
Auxiliary information 
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Table 5 

Counts of Observed Response Patterns 
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Taljle 5 



Item Parameter Estimates: Undifferentiated 
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Table 6 

Item Parameter Estimates: Population Differentiated 
with Respect to Factor A Only 

- ^ -Relative ^ ^ Relative 

Item b SE(b) Efficiency — a --^E(a) Efficiency 

1 -.436 .062 .875 .869 .069 6.142 

2 -.217 .077 .874 .622 i054 3.030 

3 .189 i072 1.114 .676 .056 2.939 

4 .465 .069 1.344 .775 .061 3.493 

Geometric average 

relative erficiency: 1.035 3.718 

Subpopolation means: .296 i -.511 

Subpopulation standard deviations: i966, ^850 
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Table 7 

Iteti Parameter Estimates: Population Differentiated 
with Respect to Factor B Only 

^ Relative ^ ^ Relative 

Item b ^ ^ciency a SE4a^— Efficiency 

1 -.408 .057 ..035 .941 .073 5.487 

2 -.211 .077 .874 .621 .056 2.818 

3 .185 .071 1.146 .686 .058 2.740 

4 .431 .064 1.563 .842 .067 2.895 

Geometric average 

relative efficiency: 1.128 3.328 

Subpbpulatibh means: .136^ -.147 

Subpbpulatibn standard devlatlbhs: 1.021^ .955 
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Table 8 

Item Parameter Estimates: Popaiation Differentiated 



with Respect to Factors A and B 
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501 North Dixon Street 

P. G. Box 3107 

Portland. OR 97209-3107 

Dr. WilliamKbch 

University of Texas-Austin 
Measurement and Evaluation 

Center 
Austin. TX 78703 

Dr. Leonard Krdeker 

Navy Personnel R4D Center 

San Diego. CA 92152 



Dr. Michael tevine 
Educational Psychology 
210 Education Bldg. 
University of Illinois 
Champaign. Ih 61801 



Dr. Charies-tewis 

Facultfiit Sociaie Wetenschappen 

Ri jksuniversiteit Groningen 

dude Boteringestraat 23 

9712Ge Groningen 

The NETHERLANDS 

Dr . Robert Linn 

College of Education 
University of Illinois 
Urbana. it 61801 



br^ Robert Ldcknsan 
Center for Naval Analysis 
itflOl Ford Avenue 
P.O. Box 16268 
Alexandria, VA 22302-0268 

Dr. FredericM. Lordj 
Educational Testing Service 
Princeton. NJ 085^41 

Dr . James tumsderi . _ _ 
Department of Psychology 
University of Western Australia 
Nedlands W.A. 6009 
AUSTRALIA 
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Dr. William L. Malby 
Chief of Naval Education 

and Training 
Naval Air Station : 
Pehsacola. FL 32508 

Dr ii Gary. Marco 
Stop 31-E 

Educational Testing Service 
Princeton. NJ 08*151 

Dr. Clessen Martin 
Army Research Institute 
5001 Eisenhower Blvd. 
Alexandria. VA 22333 



Dr. James McBride 
Psychological Corporation 
c/o Harcourt. Brace, 

Javanovich Inc. 
1250 West 6th Street 
San Diego, CA 92101 

Dr. Clarence McCbrrhicR 

HQ^MEPCOM 

MEPCT-P - 

2500 Green Bay Road 

North Chicago. IL 60064 

Mr; Robert Mckinley 

University of Toledo 

Department of Educational Psychology 

Toledo, OH **3606 

Dr . Barbara Means 

Human Resources 

Research Organization 
lido South Washington 
Alexandria, VA 22314 

Dr._Hobert Mislevy 
Educational Testing Service 
Princeton. NJ 08541 

Headiqjuarters. Marine Corps 
Code MPI-20 -_ 
Washington. DC 20380 

Dr. W. Alan Nicewahder 
University of Oklahoma^ 
Department of Psychology 
Oklahoma JSity. OK 73069 
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Dr; William E. Nord brock 
FMC-ADCb Box ?5 
APOi NY 097 in 

Dr. Meiyin R. Nbyick 
356 Lindquist Center 

for Heasarement 
University of Iowa 
Iowa City, lA 52242 

Director « Manpower and Personnel 

Laboratory, 

NPRDC (Code 06) 
San Diego, CA 92152 



Library, NPRDC 

Code P20JL 

San Diego, CA 92152 

Command ihg Officer* 

. Naval Research Laboratory 

Code 2627 

Washington i DC 20390 



Dr. James Olson 
WiCAT, inc. 

1875 South State Street 
Orem. UT 8Ub57 

Office Of Naval Research, 

Code 1112PT 
800 N. Quincy Street^ 
Arlihgtbh, VA 22217-5000 
(6 Copies) 

Special Assistant for Marine 

Corps Hatters j 

0NR Code OOMC 
800 N^ Quincy St. 
Arlington, VA 22217-5000 

Dr^ Judith Orasanu 
Army Research Institute 
5001 Eisenhower Avenue 
Alexandria, VA 22333 

Wayne M^ Patience 
American Council on Education 
GED Testing Service, Suite 20 
One Dupbht Circle, NW 
Washington. DC 20036 
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Dr. James Paulson 
Department of Psychology 
Portland State University 
P.O* Box 751 
Portland. OR 97207 

Dr. Roger Penneil 

Air Force Hiiiman Resources 

Laboratory 
Lowry AFB, CO 80230 

Dr^, Hark D. Reckase 
ACT 

P. 0. Box 168 : 
Iowa City, lA 52243 

Dr* Malcolm Ree 
AFHRfc/MP 

Brooks AFB. TX 78235 

Dr. Carl Ross 
CNET^PDCD 
Building go 

Great takes NTC, XL 60088 

Dr^ J. Ryan _ __ „ 
Department of Education 
University of South Carolina 
Columbia, SC 29208 



Dr. Fumiko Samejlroa 
Department of Psychology 
University of Tennessee 
Khoxville. TH 37916 

Hr^Df ew Sands 

NPRDC Code 62 

San Diego, CA 92152 



Dr* Robert Sasmor 
HQDA DAMA-ARt- - 
Pentagon, Room 3E516 
Washington, DC 20310-0631 
USA 

Dr. Mary Schratz 

Navy Personnel R&D Center 

San Diego. CA 92152 

Pr^ 9. Steve Sellman 
bASD(MRA4t) --^ 
2B269 the Pentagon 
Washington I DC 20301 
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Dri Kazuo Shigemasu 
7-9-2*1 kugenuma-kaigan 
Fujusawa 251 
JAPAN 

Dr. Wiliiam Sims _-_ 
Center for Navai Analysis 
M^dl Ford Avenue 
P.O. Box 16268 - - 
Alexandria. VA 22302^0268 

Dr. H. Wallace Sinaikd 
Manpower Research 

and Advisory Services 
Smithsonian Institution 
801 North Pitt Street 
Alexandria. VA 2231« 

pr . Richard Sorensen 
Navy Personnel RSD Center 
San Diego. CA 92152 

Dr, PaulSpeckman 

University of Missouri _ 
Department of Statistics 
Columbia. MO 65201 

Dr. Martha Stocking 
Educational Testing Service 
Princeton. NJ 085^1 

Dr, Peter Stoloff^ ^ 
Cehterfbr Navai Analysis 
§00 North Beauregard Street 
Alexandria. VA 22311 



Dr. William Stout 
University of Illinois 
Department of Mathematics 
Urbana. IL 5T801 



Maj. Bill Strickland 

AF/MPXbA 

ME 168 Pentagon 

Washington. DC 2033Q 

Dr. Hariharan Swamlbathan^ 
Laboratory of Psychometric and 

Eval uatibnResearch 

School of Education 
University of Massachusetts 
Amherst. MA 01003 ' 
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Mr. Brad Syrripsbh. 

Navy Personnel R&D Center 

San Diego. CA 92152 



br •kikumi Tatsuoka 
CERL 

252 Engineering Research 

Laboratory 
Urbaha. IL 61801 

Dr. Maurice tatsuoka 
220_Educatioh Bldg 
1310„S, Sixth St. 
Champaign. IL 61820 

Dr^ David Thissen. 
Departmeht of Psychology 
University of Kansas 
Lawrence. KS 66044 

Mr. Gary Thomasson 
University of_Iliinois 
Educational Psychology 
Champaign. IL 61820 

Dr. Robert tsutakawa 
The Fred Hytchihsph_ 

Cancer Research Center 
Division of Public Health Sci. 
1124 Columbia Street 
Seattle. WA 98104 

Dr. Ledyard Tucker 
University of Illinois 
Department of Psychology 
603 E. Daniel Street 
Champaign. IL 61820 

Dr^^Vern W* Urry^^ _ 
Personnel R&D Center 
Office of Personnel Management 
1900 E. Street, NW 
Washington. DC 20415 

Dr. David Vale ^ 
Assessment Systems Corp. 
2233 University Avenue 
Suite 310 

St. Paul. MN 55114 

Dr. Frank Vicino:^ 

Navy Personnel R4D Center 

San Diego, CA 92152 
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Dr. Howard Walner 
Division of Psychological Studies 
Educational Testing Service 
Princeton, NJ be5^^ 

br • Ming«F4ei Wang 
tindquist Center 

for Measurement 
University of Iowa 
Iowa City, lA 522212 

Dr . Thomas- A. Warm _ - 
Coast Guard Institute 
P. 0* Substation 18 
Oklahoma City, OK 73169 

Dr . Brian Waters 
Prbgratrj Manager 
Manpower Analysis Program 
HuiTiRRO 

noe s. Washington St. 
Alexandria, VA 2231'* 

Dr. David- J* Weiss 
N650 Elliott Hall 
University of Minnesota 
75 E. River Road _ _ 
Minneapolis, MN 55^55 

Dr.: Ronald Ai Weitzmari 
NPS, Code 5^W2 
Monterey, CA 92152 

Major John Welsh 
AFHRL/MOAN _ ^ 
Brooks AFB, TX 78223 

Dr* Band R. Wilcox 
University of Southern 

California 
Department of Psychology 
Los Angeles, CA 90007 

German Military Representative 
ATTN: Wolfgang Wildegrube 

Streitkraefteamt 

D-530D-Bonn 2 _ 
UQQQ Brandywine Street, NW 
Washington, DC 20016 



Dr. Bruce Williams 
Department of Educational 

Psychology 
University of Illinois 
Urbana, IL 61801 

Dr. Hilda Wing 
Army Research-Institute 
5001 -Eisenhower Ave. 
Alexandria, VA 22333 

Dr. Martin F. Wiskoff 

Navy Personnel R&D Center 

San Diego, CA 92152 

Mr. John H. Wolfe 

Navy Personnel RiD Center 

San Diego, CA 92152 

Dr. George Wong 
Biostatistics Laboratory 
Memorial Slbari-Kettering 

Cancer Center 
1275 York Avenue 
New York, NY 10021 

Dr> Wendy Yen 
CTB/HcGraw Hill _ 
Del Monte Research Park 
Monterey, CA 939'«0 
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